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1.  Introduction 

A  high-resolution  process  timing  facility,  called  HRTIME,  has  been  implemented  for  the 
Cedar  system.  HRTIME  is  an  extension  of  the  Concentrix®  USER  and  SYSTEM  process  time 
measurements.  It  times  both  execution  and  non-execution  process  states  with  10  /usee  accuracy. 
In  addition,  HRTIME  provides  individual  processor  timing  measurements  to  give  a  detailed 
account  of  the  time  spent  in  various  states  of  sequential  and  concurrent  execution. 

The  main  purpose  of  this  manual  is  to  explain  how  to  use  the  HRTIME  facility.  In  particu¬ 
lar,  the  manual  discusses  how  to  access  the  timing  data,  to  correctly  time  a  program  section,  and 
to  interpret  the  resulting  time  measurements.  Although  a  brief  overview  is  given  describing  what 
the  time  measurements  are  and  how  they  are  produced,  the  user  should  refer  to  [BELM87]  for  a 
complete  discussion  of  the  HRTEME  design  and  implementation. 
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2.  HRTIME  Design  Overview 

The  goal  of  the  HRTIME  facility  is  to  provide  high-resolution,  detailed  timing  measure¬ 
ments  of  process  operation.  HRTIME  is  based  on  measured  process  timing  with  all  times  meas¬ 
ured  at  10  /isec  resolution  [Malo86],  The  Concentrix  USER  and  SYSTEM  process  states  are 
extended  to  four  execution  states,  and  the  measured  times  spent  in  these  states  are  kept  on  a  per 
processor  basis.  Non-execution  states  are  also  defined  to  track  the  time  a  process  spends  ready, 
blocked  or  idle. 


2.1.  Execution  States 


Four  execution  states  are  defined  by  HRTIME :  USER,  SYSTEM,  OVERHEAD,  and  KER¬ 
NEL.  The  USER  state  is  active  when  user  code  is  being  executed.  Processing  of  system  calls 
occurs  in  the  SYSTEM  state.  Interrupt  processing  that  can  be  attributed  to  the  current  process 
falls  into  the  OVERHEAD  state;  this  includes  interrupts  for  page  faults  and  general  exceptions, 
such  as  a  floating  point  exception.  Interrupts  not  directly  associated  with  the  current  process  are 
processed  in  the  KERNEL  state;  cross-processor  interrupts,  device  interrupts  and  timer  inter¬ 
rupts  are  considered  part  of  the  KERNEL  state.  Actually,  the  KERNEL  state  is  not  a  "process” 
execution  state  at  all,  but  a  situation  where  the  operating  system  itself  is  executing.  For  this  rea¬ 
son,  only  the  USER,  SYSTEM  and  OVERHEAD  states  are  timed  for  a  process. 

In  a  multiprocessor  system  such  as  Cedar,  it  is  possible  for  a  process  to  be  executing  sequen¬ 
tially  on  one  processor  or  concurrently  on  several  processors.  To  further  audit  execution  time, 
HRTEME  keeps  time  measurements  on  a  processing  resource  basis.  In  Cedar,  a  sequential  process 


executes  on  an  IP,  a  detached-CE,  and/or  one  CE  of  a  cluster.  HRTIME  maintains  a  USER,  , 

SYSTEM  and  OVERHEAD  timer  for  each  of  these  "sequential"  processing  resources  as  part  of  j 

the  overall  process  time  measurements.  j 

All  concurrent  processes  execute  on  the  computational  complex  of  the  Alliant  FX/81.  How-  < 

ever,  it  is  possible  for  processors  participating  in  a  concurrent  computation  to  be  in  different  , 

states  of  execution.  For  this  reason,  the  execution  states  should  really  be  monitored  at  the  pro-  | 

cessor  level.  For  concurrent  processes,  HRTIME  measures  execution  times  per  processor  per  j 

state2.  USER,  SYSTEM  and  OVERHEAD  timers  are  defined  for  each  of  the  eight  CEs. 

< 

Although  the  execution  state  timers  defined  above  give  detailed  timing  information,  a  com-  \ 

plicated  calculation  must  be  made  to  determine  total  elapsed  time,  especially  in  the  case  of  a  con-  I 

current  process.  For  this  purpose,  a  VIRTUAL  execution  state  is  defined;  when  any  processing  j 

resource  is  executing  in  USER,  SYSTEM  or  OVERHEAD  state,  the  process  is  in  a  VIRTUAL  ] 

execution  state.  A  single  timer  is  defined  for  the  process  VIRTUAL  execution  state.  1 


2.2.  Non-execution  States 

In  addition  to  the  execution  states,  three  non-execution  process  states  are  recognized: 
READY,  BLOCKED,  and  IDLE.  Obviously,  when  a  process  is  ready  to  execute,  but  not 
currently  running,  it  is  in  the  READY  state.  Similarly,  a  process  is  in  the  BLOCKED  state  when 
it  is  blocked  from  execution.  The  IDLE  state  occurs  when  a  process  is  waiting  for  something  to 
do.  Because  the  process  is  not  executing,  only  one  timer  is  needed  for  each  of  these  non¬ 
execution  states. 


2.3.  State  Timing 

HRTIME  works  by  detecting  changes  in  execution  and  non-execution  states,  updating  the 
time  spent  in  the  old  state,  and  beginning  the  measurement  of  the  delta  time  in  the  new  state. 
The  high-resolution,  real-time  clock  is  used  to  mark  the  point  in  time  of  the  beginning  of  a  new 
state  and  the  end  of  the  old  state.  Simple  calculations  are  then  made  to  update  the  old  state 
times  to  their  new  values.  For  further  description  of  HRTIME  implementation  see  [BELM87]. 

3.  HRTIME  Data  Structures 

The  HRTIME  facility  is  always  enabled.  The  operating  system  maintains  the  time  meas¬ 
urements  as  part  of  the  process’s  state.  The  state  timers  are  stored  as  64-bit  integer  values  indi¬ 
cating  the  number  of  10  //sec  time  units  measured.  The  timer  data  structures  are  allocated  as 
part  of  a  larger  process  measurement  structure  in  the  process’s  read-only  address  space.  This 
allows  reference  to  any  of  the  timers  directly  from  the  user’s  program. 

The  process  timing  data  structures  are  declared  in  <sys/csrdetc.h>.  The  C  data  structure 
definitions  of  interest  for  this  document  are  reproduced  below3: 

1  A  process  is  said  to  be  concurrent  if  it  requires  more  than  one  CE  during  its  execution 

1  This  is  not  done  by  Concentrix.  Whenever  Concentrix  detects  any  processor  to  be  in  SYSTEM  state,  the  whole 
process  is  considered  to  be  in  SYSTEM  state.  USER  time  accumulates  only  when  all  processors  are  in  USER  state 

1  The  hreval  structure  contains  two  32— bi t  unsigned  integers  which  together  form  a  single  64-bit  unsigned  in¬ 
teger 


June  26,  1987 


„  M*  IS 


■it'.U 


3 


struct  exectime 

{  struct  hrcval  user;  /*  user  time  */ 

struct  hrcval  system;  /*  system  time  * / 

struct  hrcval  overhead;  /*  overhead  time  */ 

> 

struct  crestime 

{  struct  exectime  cce[>,];  /*  CE  times  */ 

struct  exectime  ip;  /*  IP  times  */ 

struct  exectime  dee;  /*  detached  CE  times  */ 

struct  exectime  cl;  /*  cluster  time  */ 

> 

struct  hrtimers 

{  struct  hrcval  pvtime;  /*  process  virtual  time  */ 

struct  hrcval  tvtime;  /*  task  virtual  time  */ 

struct  hrcval  notready;  /*  time  not  ready  */ 

struct  hrcval  ready;  /*  time  ready,  not  running  */ 

struct  hrcval  idle;  /*  task  idle  time  */ 

struct  crestime  execution;  /*  execution  times  */ 

> 

The  hrtimers  structure  defines  all  time  values  HRTIME  keeps  for  the  process4. 

Together  with  the  process’s  time  values,  a  second  set  of  timers  is  maintained  for  the 
process’s  children  processes.  Whenever  a  child  process  of  the  current  process  completes,  it  adds 
its  children  process  times  plus  its  own  times  to  the  children  process  times  of  its  parent,  the 
current  process.  The  current  process  will  likewise  propagate  is  times  to  its  parent  when  it  com¬ 
pletes. 


4.  Using  the  HRTIME  Facility  In  A  Program 

The  HRTIME  facility  is  intended  to  be  used  is  the  same  manner  as  the  standard  Concentrix 
timing  facility.  That  is,  routines  to  access  the  HRTEME  measurements,  similar  to  calling  etimc() 
and  dtimef)  to  retrieve  USER  and  SYSTEM  times  values,  are  to  be  used  in  the  same  way  for  tim¬ 
ing  program  sections  and  measuring  overall  program  execution.  However,  unlike  normal  Concen¬ 
trix  timing,  HRTIME  provide  significantly  more  detailed  timing  information.  Interpreting  the 
HRTIME  measurements  is  the  subject  of  a  later  section.  Using  the  HRTIME  facility  is  discussed 
in  this  section  and  the  next. 


4.1.  Accessing  HRTIME  Measurements 

Although  the  timers  are  in  the  user’s  virtual  address  space,  because  the  process  states  are 
dynamically  changing,  it  can  be  difficult  to  read  a  consistent  set  of  timer  values  directly.  More¬ 
over,  the  user  will  have  to  perform  update  calculations,  which  the  operating  system  normally 
does  upon  a  state  change,  to  produce  up-to-date  time  values.  The  recommended  way  to  access 
the  HRTIME  measurements  is  to  use  the  system  call  gethrtimersf)  provided  as  part  of  the  csrd 


4  tvlime  is  discussed  in  a  later  section. 
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The  format  of  the  gcthrtimersf)  system  call  is  shown  below: 
gethrtimers (type,  hrtimes) 

int  type;  /*  HRTIMERS_SELF  (0)  */ 

/*  HRT I MERS_CHI LDREN  (1)  */ 
struct  hrtimers  ‘hrtimes; 

Being  a  system  call,  gethrtimers/ )  causes  a  change  of  state  from  USER  to  SYSTEM  on  the  calling 
processor.  In  addition,  gethrtimers()  forces  all  currently  active  timers  to  be  updated  to  a  con¬ 
sistent  state.  The  appropriate  time  values,  current  process  or  children,  are  then  transferred  to 
the  user’s  buffer,  pointed  to  by  hrtimes,  and  the  process  continues. 

Raw  timing  data  are  returned  by  gethrtimers ().  If  the  user  wants  decimal  time  values  in 
seconds,  the  csrd  library  routine  hrtsecs()  can  be  called.  The  format  of  the  hrtsecs()  function  call 


double  hrtsecs (hrtime) 
struct  hrcval  *hrtime; 


4.2.  Time  Measurements  of  a  Program  Section 

The  general  procedure  for  making  time  measurements  of  a  section  of  a  program  is  shown 
below8: 

gethrtimers (type,  hrtimesl) ; 

<execute  program  section> 
gethrtimers (type ,  hrtimes2) ; 

<calculate  the  time  difference  between  hrtimes2  and  hrtimesl> 

As  is  obvious  from  above,  the  time  to  execute  the  program  section  is  really  the  difference  between 
two  struct  hrtimers  samples,  hrtimesl  and  hrtimes2,  determined  before  and  after  the  program 
section.  Following  this  procedure,  time  measurements  for  various  program  sections  can  be  made. 
In  fact,  if  the  time  samples  are  saved,  a  time-sample  trace  can  be  kept  during  program  execution 
and  a  post-processor  used  to  calculate  the  desired  incremental  time  values. 


4.3.  Program  Compilation 

To  use  the  HRTIME  facility,  the  user  program  must  include  the  following  files: 

sys/time . h 
sys/csrdetc .  h 

In  addition,  the  user  program  must  be  compiled  with  the  csrd  library. 

*  This  document  does  not  explain  how  to  deal  with  the  consistency  problems  involved  in  directly  accessing  the 
HR  TIME  data  structures.  Further  discussion  of  this  issue  might  be  documented  at  a  later  time  depending  on  need 

'  Usually,  the  type  argument  to  gethrtimen()  will  be  HRTIMERS_SELF;  i.e.,  the  current  process. 
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5.  Automatic  Program  Time  Measurements 

In  some  cases,  the  user  will  want  to  time  a  program  as  a  whole.  A  program,  hrtime,  has 
been  written  to  run  the  user’s  program  and  print  out  timing  statistics7  without  requiring  any 
program  modification.  The  hrtime  command  format  is: 

hrtime  <user  program>  <user  program  arguments> 

The  timing  results  ,/foduced  by  hrtime  include  the  execution  and  non-execution  times.  The  exe¬ 
cution  times  ar  -  -<r  <wn  as  individual  processor  times.  An  example  of  the  hrtime  output  for  a  con¬ 
currently  executing  program  is  given  below: 

PROCESS  /  TASK  TIME 


process  virtual 

5.89677 

task  virtual 

5 . 89677 

not  ready 

0.18924 

ready 

1 . 64093 

idle 

0.10224 

IP,  DETACHED  CE  and  CLUSTER  TIME 

User 

System 

Overhead 

ip 

0.00000 

0.00000 

0 . 00000 

dee 

0.00105 

0.05764 

0.01262 

cluster 

0.00000 

0.00000 

0 . 00000 

CE  TIME 

User 

System 

Overhead 

ce  [0] 

5.73250 

0.00092 

0.00737 

ce  [1] 

5.73529 

0.00000 

0.00335 

ce  [2] 

5.73540 

0.00000 

0.00393 

ce  [3] 

5.73318 

0.00416 

0.00863 

ce  [4] 

5.73358 

0 . 00000 

0.00436 

ce  [5] 

5 . 73354 

0.00000 

0.00264 

ce  [6] 

5.73285 

0.00000 

0.00288 

ce  [7] 

5.71049 

0.02005 

0.00373 

HRTIME  and  Fortran 

Although  the  above  discussion  uses  C  for  describing  HRTIME  data  structures  and 

accessing  the  HRTIME  facility  from  Fortran  is  no  problem.  The  user  only  has  to  declare  an 
equivalent  struct  hrtimers  data  structure  whose  address  will  be  passed  to  the  gethrtimersf)  rou¬ 
tine.  The  following  shows  a  suggested  Fortran  declaration  and  a  call  to  gethrtimersf ): 

integer  type 
integer*4  hrtimes (2 , 38) 

7  A  manual  page  is  available  for  the  hrtime  program  on  the  CSRD  AJIiant  machines. 
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call  gethrtimers (type, hrtimes) 


In  this  case,  the  hrtimes  array  is  organized  as: 


process  virtual  time 
task  virtual  time 
time  not  ready- 
time  ready,  not  running 
task  idle  time 


hrtimes  (1:2, 1) 
hrtimes  (1:2,2) 
hrtimes (1:2,3) 
hrtimes (1:2,4) 
hrtimes (1:2,5) 


ce  [0] 
ce[l] 
ce  [2] 
ce[3] 
ce  [4] 
ce  [5] 
ce  [6] 
ce[7] 


user 

hrtimes  (1:2,6) , 
hrtimes (1:2,9) , 
hrtimes  (1:2, 12) , 
hrtimes (1:2, 15) , 
hrtimes (1:2, 18) , 
hrtimes  (1:2, 21) , 
hrtimes (1:2, 24) , 
hrtimes (1 : 2 , 27) , 


system 

hrtimes  (1:2,7) , 
hrtimes  (1:2, 10) , 
hrtimes (1 : 2, 13)  , 
hrtimes (1: 2, 16) , 
hrtimes (1:2,19), 
hrtimes (1:2, 22) , 
hrtimes (1:2, 25) , 
hrtimes (1 : 2 , 28) , 


overhead 
hrtimes (1:2,8) 
hrtimes  (1:2,11) 
hrtimes (1:2, 14) 
hrtimes (1:2,17) 
hrtimes (1:2, 20) 
hrtimes  (1 : 2, 2  3) 
hrtimes  (1:2, 26) 
hrtimes (1:2,29) 


user  system  overhead 

ip  :  hrtimes  (1 : 2 , 30) ,  hrtimes (1 : 2 , 31)  ,  hrtimes (1 : 2 , 32) 

detached  ce  :  hrtimes (1 : 2 , 33) ,  hrtimes (1 : 2 , 34) ,  hrtimes (1 : 2 , 35) 
cluster  :  hrtimes (1 : 2 , 36) ,  hrtimes  (1 : 2 , 37) ,  hrtimes (1 : 2 , 38) 


As  before,  all  times  are  64-bit  unsigned  integer  values  indicating  the  number  of  10  psec 
time  units  measured.  Again,  the  hrtsecs()  function  can  be  called  to  convert  the  integer  time 
values  to  floating  point  values. 


7.  HRTIME  and  Xylem 

Xylem  supports  multitasking  of  a  process  for  concurrent  execution  across  multiple  Cedar 
clusters.  Concentrix  runs  on  each  individual  cluster  and  schedules  Xylem  tasks  for  execution. 
The  question  is  how  the  HRTIME  facility  performs  time  measurements  of  Xylem  tasks.  Because 
Xylem  tasks  are  essentially  equivalent  to  Concentrix  processes,  the  time  measurement  mechan¬ 
ism  is  exactly  the  same.  Hence,  all  previous  discussion  also  applies  to  Xylem  tasks.  However, 
some  additional  HRTIME  measurement  mechanisms  for  Xylem  processes  are  provided. 

A  Xylem  process  is  made  up  of  one  or  more  tasks.  A  Xylem  process  represents  a  logical  exe¬ 
cution  environment;  that  is,  only  the  tasks  actually  execute.  However,  HRTJME  also  measures 
time  for  a  Xylem  process.  This  is  done  by  adding  the  execution  time  results  of  the  Xylem 
process’s  tasks  to  its  HRTIME  time  totals  when  the  tasks  complete.  Thus,  the  execution  time 
measurements  for  a  Xylem  process  are  the  sum  of  all  its  tasks’  execution  times. 

For  a  normal  Concentrix  process,  the  process  VIRTUAL  time,  pvtime,  is  equivalent  to  the 
task  VIRTUAL  time,  tvtime.  However,  for  Xylem  tasks  and  processes,  these  two  VIRTUAL 
times  take  on  different  meanings.  Task  VIRTUAL  time  for  a  Xylem  task  is  analogous  to  process 
VIRTUAL  time  for  a  Concentrix  process.  That  is,  task  VIRTUAL  time  accumulates  whenever 
any  processing  resource  used  by  the  task  is  in  USER,  SYSTEM  or  OVERHEAD  state.  Xylem 
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tasks  do  not  use  the  process  VIRTUAL  timer. 

On  the  other  hand,  Xylem  processes  do  use  the  process  VIRTUAL  timer  for  determining 
total  elapsed  execution  time  for  the  Xylem  process.  A  Xylem  process  is  in  a  VIRTUAL  execution 
state  whenever  any  of  its  tasks  are  executing  in  USER,  SYSTEM  or  OVERHEAD  state;  this  is 
analogous  to  the  Concentrix  process  VIRTUAL  state  definition  except  on  at  higher  level. 


8.  Interpreting  HRTIME  Measurements 

HRTIME  provides  significantly  more  detailed  timing  information  than  the  standard  Con¬ 
centrix  USER  and  SYSTEM  times.  In  fact,  in  nearly  all  cases,  the  HRTIME  facility  can  take  the 
place  of  Concentrix  timing  for  process  time  measurements;  certainly,  HRTIME  will  be  used  for 
non-execution  timing  and  Xylem  process/task  timing.  However,  together  with  the  benefit  of 
more  information  comes  the  problem  of  understanding  what  it  all  means.  This  section  makes  a 
few  comments  on  interpretating  HRTIME  measurements.  An  all-inclusive  discussion  is  impossi¬ 
ble  since  the  time  measurements  will  be  used  for  different  purposes.  However,  as  users  gain  more 
experience  with  HRTIME,  they  will  become  more  confident  in  their  understanding  of  the  meas¬ 
urements,  and  further  consensus  on  standard  timing  measurement  methodology  is  expected. 

One  aspect  of  HRTIME  that  might  be  curious  is  why  the  definition  of  USER,  SYSTEM  and 
OVERHEAD  states  instead  of  staying  with  USER  and  SYSTEM.  Some  system  processing  actu¬ 
ally  occurs  on  behalf  of  the  user  and,  therefore,  should  be  measured  separately  from  the  overhead 
of  general  OS  operations.  Doing  so  provides  the  user  with  a  measurement  of  how  much  overhead 
processing  the  program  is  really  experiencing  during  its  execution  and  also  how  much  system  sup¬ 
port  the  program  is  requiring.  By  monitoring  the  OVERHEAD  times,  as  well  as  the  USER  and 
SYSTEM  times,  the  user  can  get  a  sense  of  how  vulnerable  the  program’s  performance  is  to  the 
overhead  processing. 

Non-execution  time  measurements  might  be  regarded  as  superfluous  by  some  users.  How¬ 
ever,  these  times  have  real  meaning  and  can  reflect  interesting  process  operation  behavior.  For 
instance,  the  READY  time  is  a  good  indication  of  the  amount  of  waiting  a  process  experiences  in 
the  scheduling  queue.  The  BLOCKED  time  is  even  more  versatile  in  that  it  encompasses  all 
dependent  waiting  (blocking)  time  encountered  by  the  process.  This  time  not  only  includes 
blocking  due  to  I/O  operations  but  also  represents  waiting  due  to  inter-process  and  inter-task 
synchronization.  IDLE  time  is  particularly  interesting  for  Xylem  tasks  because  it  can  be  used  to 
compute  a  task  utilization  measure  indicating  the  percentage  of  time  a  task  was  executing  some 
portion  of  the  user’s  program. 

For  the  most  part,  the  execution  time  measurements  have  clear  definitions;  e.g.,  the  time 
spent  executing  in  SYSTEM  state  on  CE  5  or  the  total  process  virtual  time.  The  complication 
comes  when  trying  to  work  back  from  the  measurements  to  what  the  program  is  actually  doing. 
For  sequential  processes,  the  HRTIME  measurements  are  not  too  difficult  to  understand.  The 
breakdown  across  the  different  sequential  processing  resources  is  interesting  because  it  shows  how 
the  process  was  scheduled  during  its  execution.  To  determine  the  total  USER,  SYSTEM  and 
OVERHEAD  times  the  user  could  sum  the  IP,  detached  CE  and  cluster  values.  However,  when 
the  execution  takes  place  on  different  physical  processing  resources,  as  in  this  case  IPs  and  CEs, 
the  user  may  want  to  regard  the  execution  times  differently. 

The  HRTIME  measurements  for  concurrent  processes  are  in  some  ways  easy  and  in  other 
ways  difficult  to  interpret.  On  the  one  hand,  the  measurements  are  just  a  simple  extension  of  the 
one  CE  cluster  execution  times  to  multiple  CEs.  On  the  other  hand,  the  actions  of  multiple  CEs 
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have  to  be  determined  instead  of  just  one.  The  goals  of  the  CE  execution  time  measurements  is 
to  give  some  indication  of  CE  resource  usage.  Ideally,  a  single  global  state  space  would  be 
defined  where  each  point  describes  a  different  combination  of  the  CE  execution  states.  Time 
spent  in  each  global  state  could  then  be  measured.  However,  the  implementation  of  this  meas¬ 
urement  model  is  impractical.  Thus,  the  individual  CE  measurements  are  provided. 

Using  the  individual  CE  measurements,  it  is  difficult  to  determine  the  amount  of  time  CE  0 
is  in  USER  state  when  CE  1  in  SYSTEM  state,  and  so  on,  with  the  other  CE  state  combinations. 
However,  it  is  unclear  whether  such  a  time  value  has  much  meaning.  Because  the  computational 
complex  is  assigned  to  a  process  as  a  single  resource,  it  is  more  important  how  the  individual  CEs 
themselves  are  utilized.  The  HRTIME  measurements  show  this  as  a  breakdown  between  execu¬ 
tion  states  for  each  CE.  Unfortunately,  HRTIME  is  unable  to  detect  when  a  CE  is  idling.  Other¬ 
wise,  the  time  spent  in  a  non-execution  state  could  be  measured  for  each  CE  and  complete  CE 
utilization  statistics  calculated.  As  it  is,  when  a  CE  is  idling,  the  CE  appears  to  be  in  USER 
state  and  USER  time  is  being  accumulated. 


9.  Conclusion 

The  HRTIME  facility  is  a  new  tool  for  timing  programs  on  the  Cedar  multiprocessor.  It  is 
possible  that  significantly  more  questions  will  be  raised  about  the  HRTIME  facility  than  were 
answered  above.  Being  a  new  tool,  this  is  to  be  expected.  However,  the  additional  detail  of  the 
HRTIME  measurement  should  be  a  benefit  over  the  simple  USER  and  SYSTEM  times  currently 
reported  by  Concentrix.  As  user  experience  with  HRTIME  grows,  it  is  hoped  that  standard  tim¬ 
ing  practices  will  be  developed. 
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